跳转至

Max-Q 选择性模仿：用于人机协同在线机器人学习

文章背景与核心概要

在现实世界的机器人应用中，人机协同（Human-in-the-Loop, HIL）在线强化学习面临着双重挑战：既要能够快速吸收人类干预以纠正错误，又要能在人类先验的基础上实现超越人类水平的自我提升。现有的方法往往在处理干预轨迹时存在信用分配不均或分布偏移的问题，导致学习效率低下。

本文提出了一种创新的训练方法，通过“MC Q-chunk Critic”和“Max-Q 选择性模仿”两个核心组件解决了上述难题。MC Q-chunk Critic 通过蒙特卡洛回报直接对动作块进行评估，确保了人类干预轨迹的有效信用分配；而 Max-Q 选择性模仿则采用“赢家通吃”策略，自动在学习人类干预与自主策略优化之间切换。实验证明，该方法在 USB 插拔等真实任务中，仅需 30 分钟即可达到 99% 的成功率，显著优于现有的 HIL-SERL 等基准方法。

📌 摘要 (Summary)

本文介绍了一种用于现实世界机器人人机协同（HIL）在线强化学习的新型训练方法。该方法旨在解决在快速吸收人类干预的同时，实现超越人类表现的性能提升这一双重挑战。

This paper introduces a novel training method for Human-in-the-Loop (HIL) online reinforcement learning in real-world robotics. The approach addresses the dual challenge of quickly absorbing human interventions while improving beyond human performance.

该方法构建于两个核心组件之上： 1. MC Q-Chunk Critic：将块级（chunk-level）动作价值回归到来自回放缓冲区的蒙特卡洛回报上，确保干预轨迹获得直接的信用分配，而非被标准的 TD 备份所稀释。 2. Max-Q 选择性模仿：通过一种硬性的“赢家通吃”规则更新执行器（actor），在当前策略和缓冲区样本之间选择 \(Q\) 值更高的动作。这使得学习过程能够自动在整合人类干预与自主自我提升之间转换，同时减轻了执行时的分布偏移。

The method is built upon two core components: 1. MC Q-Chunk Critic: Regresses chunk-level action values onto Monte Carlo returns from the replay buffer, ensuring intervention trajectories receive direct credit rather than being diluted by standard TD backups. 2. Max-Q Selective Imitation: Updates the actor using a hard winner-take-all rule that selects the higher-\(Q\) action between the current policy and a buffer sample. This automatically transitions the learning process between incorporating human interventions and autonomous self-improvement while mitigating execution-time distribution shift.

关键结果

  • 真实世界任务：在仅使用 20 次演示的 USB 拾取与插入任务中，ACT QChunk-MCBC 变体在 30 分钟的 HIL 训练内实现了 99% 的成功率（相比之下，HIL-SERL 需要约 5 小时）。
  • 仿真基准：在 Peg Insertion（插销）和 Square（方形）任务中，ACT/Flow Q-chunk 变体在约 30 分钟内达到了 \(\ge\)96% 的成功率，在成功率-时间前沿上超越了 HIL-SERL、EXPO 和 E2HiL 等现有基准。

Key Results

  • Real-World Task: On a USB pick-and-insertion task using only 20 demonstrations, the ACT QChunk-MCBC variant achieves a 99% success rate within 30 minutes of HIL training (compared to ~5 hours for HIL-SERL).
  • Simulation Benchmarks: On Peg Insertion and Square tasks, ACT/Flow Q-chunk variants achieve \(\ge\)96% success within roughly 30 minutes, outperforming existing baselines such as HIL-SERL, EXPO, and E2HiL on the success–time frontier.

📄 论文摘要 (Abstract)

用于真实机器人的人机协同（HIL）在线强化学习必须能够快速吸收人类干预，同时继续超越人类先验进行提升。我们针对此场景提出了一种基于两个组件的训练方法。首先，\emph{MC Q-chunk} 评论家（critic）将块级动作价值回归到来自回放缓冲区的蒙特卡洛回报上，执行样本平均（行为）策略评估，从而使干预轨迹获得直接的信用分配，而不是被当前策略的 TD 备份所稀释。其次，\emph{max-Q 选择性模仿} 通过在每个状态下模仿当前策略动作与缓冲区样本中 \(Q\) 值较高的动作来更新执行器，并采用硬性的“赢家通吃”规则。该规则自动在从干预中学习和策略内自我提升之间切换：当自主策略更强时，目标与策略分布对齐，从而减少了原本会导致执行时分布偏移的策略-目标样本差距。在实践中，我们使用标准的评论家集成均值对候选动作进行评分以减少比较噪声，而无需软化目标或引入分数差距阈值。在具有 20 次演示的真实 USB 拾取与插入任务中，ACT QChunk-MCBC 在 30 分钟的 HIL 训练内达到了 99% 的成功率，而 HIL-SERL 需要约 5 小时才能收敛。在 Peg Insertion 和 Square 的仿真中，ACT/Flow Q-chunk 变体同样在约半小时的有效训练内达到了 \(\ge\)96% 的成功率，在成功率-时间前沿上优于 HIL-SERL、EXPO 和 E2HiL。

Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an \emph{MC Q-chunk} critic regresses chunk-level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, \emph{max-Q selective imitation} updates the actor by imitating, at each state, the higher-\(Q\) action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy--target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk-MCBC attains 99\% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach \(\ge\)96\% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success--time frontier.

🔗 链接与资源

许可信息：知识共享署名 4.0 国际许可协议 license icon